Fruit quantity measurement system and method

ABSTRACT

A fruit quantity measurement system includes an unmanned aerial vehicle configured to receive GPS information about a fruit tree region of a predetermined area and an RF signal transmitted from an RF transmitter installed for each fruit tree and provide fruit tree images captured using at least one image sensor while flying based on a predetermined flight plan over the fruit tree region, and a monitoring server configured to be connected to the unmanned aerial vehicle through a communication network to receive GPS information and an RF signal from the unmanned aerial vehicle, match and store location data for each fruit tree, analyze the fruit tree image, measure the number of fruits for each fruit tree, store fruit counting information, and provide the stored location data or fruit counting information for each fruit tree according to a request from a user terminal authorized in advance.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims is a national phase of International ApplicationNo. PCT/KR2022/003233 filed on Mar. 8, 2022, which claims priority toKorean Patent Application No. 10-2021-0033932 filed on Mar. 16, 2021,the entire contents of which are herein incorporated by reference.

This patent is the result of a study conducted with the support of theKorea Institute for Advancement of Technology with financial resourcesfrom the Korean government (Ministry of Trade, Industry and Energy) in2021 (Unique Project Number 1415176969, Detailed Project Number:P0014718, Project name: Smart farming demonstration spread project using5G-based drones).

This patent is the result of a study conducted with the support of theKorea Evaluation Institute of Industrial Technology with financialresources from the Korean government (Ministry of Trade, Industry andEnergy) in 2022 (Unique Project Number 1415181149, Detailed ProjectNumber: 20018828, Project name: Development of optical filter forwavelength control and light source module for lighting device).

BACKGROUND 1. Field

The present invention relates to a fruit quantity measurement techniquefor measuring the number of fruits using a fruit tree image capturedfrom top of a fruit tree.

2. Description of Related Art

The contents described in this Background Art merely provide backgroundinformation on the present embodiment and do not constitute the relatedart.

Fruit counting of fruit trees is important information for producers todecide their intention to ship and to predict supply and demand. As afruit counting method of fruit trees, sample survey and monitoringsurvey data are mainly used, so there is a high possibility that asubjective factor of a counter will be included.

As an example, as a method of predicting production, there is a methodof calculating production by multiplying production (yield) per unit ofmature tree area by a mature tree area. In this case, data announced bythe National Agricultural Products Quality Management Service are usedfor the mature tree area, and data from sample farms and monitoringsurveys are used to estimate the number of mature trees and production.If necessary, weather information such as temperature, sunlight, andprecipitation may be additionally reflected. However, since the samplesurvey and monitoring survey are conducted directly by manpower, thereis a problem that the surveys are inaccurate due to human subjectivityand the possibility of information change caused by meteorologicaldisasters, diseases and pests, and the like.

According to the data of the National Agricultural Products QualityManagement Service, the number of crops fluctuates every year due toweather damage such as typhoons and pests, and information on the numberof fruit trees and fruits is bound to be inaccurate due to these weatherdisasters and pests.

Agricultural production information is collected from the NationalAgricultural Products Quality Management Service's ‘Main Crop ProductionTrend’ and the Rural Development Administration's ‘Agricultural andLivestock Product Income Data Book’, and data on the ‘number of maturetrees’ are derived from the collected information. However, there is thepossibility of an error in a reference value due to a dense orchard,that is, an increasing trend in the number of adult trees per unit area.

In addition, the method of calculating the number of fruits in a fruittree recognizes a fruit region using color in a still image obtained bycapturing a fruit tree with fruits such as apples and tangerines, andcalculates the number of fruits using the recognized fruit region. Themethod of calculating the number of fruits using still images has beenmainly used for research purposes, such as calculating the number offruits for one fruit tree.

Therefore, in the case of trying to calculating the number of fruits forhundreds of thousands of fruit trees by the conventional method ofcalculating the number of fruits using still images, only when a personshould directly capture still images for each fruit tree, the number offruits for each fruit tree should be calculated using still imagesphotographed for each fruit tree, and the number of fruits for eachfruit tree should be summed, the number of fruits can be calculated forall fruit trees in the fruit farm.

As such, the conventional method of calculating the number of fruitsusing still images has limitations in calculating fruit production onlarge-scale fruit orchards, etc., or calculating the number of fruits onfruit trees of the entire fruit farmhouse for insurance compensation incase of a disaster.

FIG. 1 is a diagram explaining a method of calculating the number offruits using a conventional image camera installed on the ground in therelated art.

Referring to FIG. 1 , in the conventional method of calculating thenumber of fruits, fruit trees and fruits are captured using two or morevisible cameras 11 and 12 on the ground, and fruit counting is performedusing fruit recognition and an angle.

In this case, similar to human eyes, the visible cameras 11 and 12installed on the ground allow a detector to receive light reflected froman object, and convert the received light into an image, therebyrecognizing the object.

Therefore, the conventional method of calculating the number of fruitsmay obtain an image of the front of the fruit trees captured by thevisible cameras 11 and 12, but the rear of the fruit tree is a blindspot in which the visible camera may not acquire an image. Accordingly,the conventional method of calculating the number of fruits has aproblem in that it is impossible to count the number of fruits in theblind spots.

In addition, in the conventional method of calculating the number offruits, in the case of a dense orchard where fruit trees are dense, thevisible cameras 11 and 12 may capture not only fruit trees to bemeasured but also the fruit trees adjacent to the fruit trees to bemeasured. As a result, there is a possibility that not only the fruittree to be measured but also the fruits of adjacent fruit trees may bemeasured during fruit counting, which may cause errors in the fruitcounting information itself.

SUMMARY

An object of one embodiment of the present invention is to provide afruit quantity measurement system and method for accurately measuringthe quantity of fruit using an RGB camera and an infrared camera mountedon an unmanned aerial vehicle.

According to an embodiment of the present invention, a fruit quantitymeasurement system includes: an unmanned aerial vehicle configured toreceive GPS information about a fruit tree region of a predeterminedarea and an RF signal transmitted from an RF transmitter installed foreach fruit tree and provide fruit tree images captured using at leastone image sensor while flying based on a predetermined flight plan overthe fruit tree region; and a monitoring server configured to beconnected to the unmanned aerial vehicle through a communication networkto receive GPS information and an RF signal from the unmanned aerialvehicle, match and store location data for each fruit tree, analyze thefruit tree image, measure the number of fruits for each fruit tree,store fruit counting information, and provide the stored location dataor fruit counting information for each fruit tree according to a requestfrom a user terminal authorized in advance.

The fruit quantity measurement system may further include: at least oneor more GPS units configured to be installed at a boundary of the fruittree region to provide GPS information.

The unmanned aerial vehicle may include one or more cameras using an RGBsensor and an infrared sensor.

The monitoring server may perform a deep learning-based fruitrecognition algorithm that extracts fruit regions from the fruit treeimage, extract features from the extracted fruit regions, and thenprovide fruit classification and prediction results based on theextracted features.

The deep learning-based fruit recognition algorithm may generate andstores training data for training a plurality of fruit imagescorresponding to big data, train a classifier by setting the features ofeach fruit, including a size, a shape, and a color of each fruit, asfruit determination criteria through the analysis of the stored trainingdata, and output the classification and prediction results including atype and quantity of fruit for the fruit tree image input by the trainedclassifier.

According to another embodiment of the present invention, a fruitquantity measurement method performed by a monitoring server thatperforms a fruit tree monitoring function for a fruit tree regionincludes: a) acquiring GPS information about the fruit tree region, andgenerating a flight plan for at least one unmanned aerial vehicle forfruit tree recognition based on the acquired GPS information; b)transmitting a flight control signal to the unmanned aerial vehicle formoving the unmanned aerial vehicle to air above a predetermined fruittree based on the flight plan of the unmanned aerial vehicle; c)identifying a predetermined fruit tree by receiving an RF signaltransmitted from an RF transmitter installed in the fruit tree throughthe unmanned aerial vehicle, and matching and storing uniqueidentification information of the predetermined fruit tree and locationdata; and d) receiving fruit tree images through the unmanned aerialvehicle, analyzing the fruit tree images, measuring the fruit quantityfor each fruit tree, and storing fruit counting information.

The fruit quantity measurement method may further include: e) providingstored location data or fruit counting information for each fruit treeaccording to a request of a user terminal authorized in advance.

The unmanned aerial vehicle may include one or more cameras using an RGBsensor and an infrared sensor, and provide a fruit tree image includingan RGB image and an infrared image through the camera.

In step d), the deep learning-based fruit recognition algorithm mayextract fruit regions from the fruit tree image, extract features fromthe extracted fruit regions, and then provide fruit classification andprediction results based on the extracted features.

The deep learning-based fruit recognition algorithm may generate andstores training data for training a plurality of fruit imagescorresponding to big data, train a classifier by setting the features ofeach fruit, including a size, a shape, and a color of each fruit, asfruit determination criteria through the analysis of the stored trainingdata, and output the fruit classification and prediction resultsincluding a type and quantity of fruit for the fruit tree image input bythe trained classifier.

A monitoring server includes: a communication unit configured to performa communication function with an unmanned aerial vehicle flying over afruit tree region or a user terminal authorized in advance; a memory inwhich a program for performing a fruit quantity measurement method forthe fruit tree region is recorded; and a processor configured to executethe program, in which the processor is connected to the unmanned aerialvehicle through a communication network by execution of the program toreceive GPS information and an RF signal from the unmanned aerialvehicle, match and store location data for each fruit tree provided fromthe unmanned aerial vehicle, analyze the fruit tree image, measure thenumber of fruits for each fruit tree, store fruit counting information,and provide the stored location data or fruit counting information foreach fruit tree according to a request from a user terminal.

Finally, the present invention may provide a computer readable recordingmedium on which a program for performing the fruit quantity measurementmethod is recorded.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram explaining a method of calculating the number offruits using a conventional image camera installed on the ground in therelated art.

FIG. 2 is a diagram explaining a configuration of a fruit quantitymeasurement system according to an embodiment of the present invention.

FIG. 3 is a block diagram illustrating a configuration of an unmannedaerial vehicle according to an embodiment of the present invention.

FIG. 4 is a front view illustrating an unmanned aerial vehicle accordingto an embodiment of the present invention.

FIG. 5 is a diagram illustrating spectral distribution of a generallong-wavelength infrared band.

FIG. 6 is a diagram illustrating main signal flows of a fruit quantitymeasurement system according to an embodiment of the present invention.

FIG. 7 is a diagram explaining a deep learning-based fruit recognitionalgorithm of a monitoring server according to an embodiment of thepresent invention.

FIG. 8 is a flowchart explaining a process of matching location data foreach fruit tree in a fruit quantity measurement method according to anembodiment of the present invention.

FIG. 9 is a diagram explaining a positioning process of an unmannedaerial vehicle according to an embodiment of the present invention.

FIG. 10 is a diagram explaining a process of identifying a fruit treeusing RF received signal strength according to distance according to anembodiment of the present invention.

FIG. 11 is an exemplary diagram illustrating a received signal strengthdistribution at a position of an unmanned aerial vehicle according to anembodiment of the present invention.

FIG. 12 is a flowchart explaining a fruit quantity measurement methodaccording to an embodiment of the present invention.

FIG. 13 is an exemplary view illustrating an infrared image acquiredthrough an unmanned aerial vehicle according to an embodiment of thepresent invention.

FIG. 14 is an exemplary diagram illustrating a process of detecting afruit area in an infrared image of FIG. 13 .

FIG. 15 is an exemplary view illustrating a fruit counting process basedon the fruit region detected in FIG. 14 .

DETAILED DESCRIPTION

The present invention may be variously modified and have severalexemplary embodiments. Therefore, specific exemplary embodiments of thepresent invention will be illustrated in the accompanying drawings andbe described in detail. However, it is to be understood that the presentinvention is not limited to a specific exemplary embodiment, butincludes all modifications, equivalents, and substitutions withoutdeparting from the scope and spirit of the present invention. Indescribing each drawing, similar reference numerals are used for similarcomponents.

Terms used in the specification, “first,” “second,” “A,” “B” etc., maybe used to describe various components, but the components are not to beinterpreted to be limited to the terms. The terms are used only todistinguish one component from another component. For example, a firstcomponent may be named a second component and the second component mayalso be similarly named the first component, without departing from thescope of the present disclosure. The term and/or includes a combinationof a plurality of related described items or any one of the plurality ofrelated described items.

It is to be understood that when one element is referred to as being“connected to” or “coupled to” another element, it may be connecteddirectly to or coupled directly to another element or be connected to orcoupled to another element, having the other element interveningtherebetween. On the other hand, it should be understood that when oneelement is referred to as being “connected directly to” or “coupleddirectly to” another element, it may be connected to or coupled toanother element without the other element interposed therebetween.

Terms used in the present specification are used only in order todescribe specific exemplary embodiments rather than limiting the presentinvention. Singular forms are intended to include plural forms unlessthe context clearly indicates otherwise. It should be understood thatterms such as “include” or “have” in this application do notpreliminarily exclude the presence or addition of features, numbers,steps, operations, components, parts, or combinations thereof describedin the specification.

Unless indicated otherwise, it is to be understood that all the termsused in the specification including technical and scientific terms havethe same meaning as those that are generally understood by those whoskilled in the art.

Terms generally used and defined by a dictionary should be interpretedas having the same meanings as meanings within a context of the relatedart and should not be interpreted as having ideal or excessively formalmeanings unless being clearly defined otherwise in the presentspecification.

In addition, each configuration, process, process, method, etc.,included in each embodiment of the present invention may be technicallyshared within a range that does not contradict each other.

FIG. 2 is a diagram explaining a configuration of a fruit quantitymeasurement system according to an embodiment of the present invention.

Referring to FIG. 2 , the fruit quantity measurement system includes,but is not limited to, an unmanned aerial vehicle 200 flying over afruit tree region 100 and a monitoring server 300 connected to theunmanned aerial vehicle 200 through a communication network.

First, an RF transmitter 110 transmitting RF signals to each fruit treeis installed in the fruit tree region 100, and at least one GPS unit 120providing GPS information GPS-A, GPS-B, GPS-C, and GPS-D is installed ateach boundary point of the fruit tree region 100 having a predeterminedarea. Here, the GPS unit 120 is a device capable of collecting locationinformation, and may be implemented as a mobile communication terminal,a beacon, an RFID tag, etc., as well as a GPS. In addition, the RFtransmitter 110 is attached to each fruit tree in the form of an RFICtag, and may transmit a unique identification number for each fruit treeas an RF signal.

A user terminal 400 for managing the fruit tree region 100 may providethe monitoring server 300 with a fruit tree arrangement map including anarea of its own fruit tree region and an arrangement state of fruittrees. The monitoring server 300 may create and store the fruit treearrangement map in the form of a map in which address information of afruit tree region, aerial photographs, and text information on thearrangement of the fruit trees are integrated, and then provide thefruit tree arrangement map to the user terminal 400.

The unmanned aerial vehicle 200 receives GPS information from the GPSunit 120 installed in the fruit tree region, receives the RF signalstransmitted from the RF transmitters 110 installed for each fruit tree,and transmits the GPS information and RF signals to the monitoringserver 300. In addition, the unmanned aerial vehicle 200 captures afruit tree image using at least one image sensor while flying in the skyabove the fruit tree region based on a predetermined flight plan, andprovides the captured fruit tree image to the monitoring server 300.

The monitoring server 300 receives the GPS information and RF signalsfrom the unmanned aerial vehicle 200, matches location data for eachfruit tree, and stores the matched location data in a database. Fruittree images are analyzed to measure the number of fruits for each fruittree, and the fruit counting information is stored in the database. Thestored location data or fruit counting information for each fruit treeis provided to the user terminal 400 according to the request of theuser terminal 400 that has been authorized in advance.

The monitoring server 300 may be a computer body for a server in ageneral sense, and may be implemented in various types of devicescapable of performing a server role. Specifically, the monitoring server300 may be implemented in a computing device including a communicationunit (not illustrated), a memory (not illustrated), a processor (notillustrated), and a database (not illustrated), and may be implementedas a smartphone, a TV, a PDA, a tablet PC, a PC, a notebook PC, otheruser terminal devices, and the like.

FIG. 3 is a block diagram illustrating a configuration of an unmannedaerial vehicle according to an embodiment of the present invention, FIG.4 is a front view illustrating the unmanned aerial vehicle according tothe embodiment of the present invention, and FIG. 5 is a diagramillustrating spectral distribution of a general long-wavelength infraredband.

The unmanned aerial vehicle 200 is a remotely controlled orself-controlled aerial vehicle without a person riding, and may beequipped with a camera, a sensor, communication equipment, or otherequipment depending on the purpose of use. The unmanned aerial vehicle200, commonly referred to as a drone, includes a remote piloted vehicle(RPV), an unmanned/uninhabited/unhumanized aerial vehicle system (UAV),an unmanned aircraft system (UAS), a remote piloted air/aerial vehicle(RPAV), a remote piloted aircraft system (RPAS), robot aircraft, etc.

Referring to FIGS. 3 and 4 , the unmanned aerial vehicle 200 includes,but is not limited to, a camera 210, a receiver 220, a communicationmodule 230 and a control module 240.

The camera 210 may be one or more cameras using an RGB sensor and aninfrared sensor. The infrared camera 211 or thermal imaging camera is acontactless measuring device that detects infrared energy and convertsthe detected infrared energy into a real image, and generates an imageusing heat rather than visible light. That is, the infrared camera 211generates an image by receiving light of an infrared wavelength band andoutputting the received light as a digital or analog image.

In general, infrared is divided into near-infrared (NIR, 0.7 to 1.4 μm),shortwavelength infrared (SWIR, 1.4 to 3 μm), mid-wavelength infrared(MWIR, 3 to 5 μm), and long-wavelength infrared (8 to 14 μm), andfar-infrared (FIR, 15 to 1000 μm).

As illustrated in FIG. 5 , the infrared camera 211 of the presentinvention may generate an infrared image targeting long-wavelengthinfrared (LWIR, 8-14 μm). A general infrared camera uses a sensor thatdoes not distinguish infrared wavelength bands, in particular,long-wavelength infrared from other wavelengths, but the infrared camera211 generates an image targeting long-wavelength infrared. Accordingly,the infrared camera 211 may capture even fruit that is covered withleaves or other obstacles, and extract temperature information of mainareas (leaves, branches, fruits, etc.) of interest of the fruit treefrom the acquired infrared image.

In this way, the long-wavelength infrared camera 211 may receiveinfrared rays emitted from a surface of a fruit tree and generate aninfrared image of a temperature distribution on the surface of theobject, thereby distinguishing an object according to radiationintensity. The radiation intensity may be calculated from the radiationenergy and area of a target object. By using the radiation curve, it canbe seen that maximum radiation intensity of the target object at atemperature of 30° C. appears at a wavelength of 10 μm.

Infrared, thermal energy, or light are all forms of energy in a categoryof the electromagnetic spectrum. However, a camera that may detectvisible light does not have the ability to detect thermal energy, but acamera that detects thermal energy may not detect visible light.

Therefore, the RGB camera 212 acquires an RGB image of each fruit treeso that the unmanned aerial vehicle 200 may be located on top of aspecific fruit tree without overlapping with neighboring fruit trees andthe infrared camera 211 may acquire infrared images for fruit countingfor each fruit tree. In this case, the fruit tree image may include anRGB image and an infrared image.

The receiver 220 includes a GPS receiver 221 and an RF receiver 222. TheGPS receiver 221 receives the GPS information so that the unmannedaerial vehicle 200 recognizes location information and moves to a fruittree region, and moves to a preset fruit tree using pre-stored locationdata. The RF receiver 222 receives uniquely identifiable RF signals foreach fruit tree transmitted from the RF transmitter 110 installed foreach fruit tree.

The communication module 230 transmits the fruit tree image captured bythe camera 210, and the GPS information and the RF signal received fromthe receiver 220 to the monitoring server 300, and receives a flightcontrol signal transmitted from the monitoring server 300.

The control module 240 may control the operations of the receiver 220and the camera 210, including flight operations (take-off, landing,attitude control, flight path determination, etc.) of the unmannedaerial vehicle 200, and perform a control operation necessary forsteering the unmanned aerial vehicle 200 based on an external flightcontrol signal.

FIG. 6 is a diagram explaining a main signal flow of a fruit quantitymeasurement system according to an embodiment of the present invention,and FIG. 7 is a diagram explaining a deep learning-based fruitrecognition algorithm of a monitoring server according to an embodimentof the present invention.

Referring to FIGS. 6 and 7 , the monitoring server 300 may receiveflight information, fruit tree images including infrared images and RGBimages, GPS information, and RF signals from the unmanned aerial vehicle200, and may generate flight control signals according to location datafor each fruit tree according to the flight information, the GPSinformation, and the RF signals and transmit the generated signals tothe unmanned aerial vehicle 200.

In addition, the monitoring server 300 may determine the flight path ofthe unmanned aerial vehicle 200 based on the pre-stored fruit treearrangement map, and generate the flight control signal so that theunmanned aerial vehicle 200 may automatically or semi-automatically flyaccording to the determined flight path and transmit the generatedflight control signal to the unmanned aerial vehicle 200.

The user terminal 400 may request location data or fruit countinginformation for each fruit tree from the monitoring server 300, and themonitoring server 300 may provide data requested by the user terminal400 in the form of various reports such as tables, time series chartsshowing time series changes, line charts, bar graphs, region chartsshowing a distribution of fruit tree regions for each region, or piecharts.

The monitoring server 300 may set fruit tree images including RGB imagesand infrared images transmitted from the unmanned aerial vehicle 200 asinput data, input the input data to a deep learning-based fruitrecognition algorithm to calculate fruit classification and predictionresults, and provide the calculated fruit classification and predictionresults to the user terminal 400.

In this case, the deep learning-based fruit recognition algorithm mayextract the fruit regions from the fruit tree images, extract featuresfor each extracted fruit region, and then provide the fruitclassification and prediction results based on the extracted features.To this end, the deep learning-based fruit recognition algorithmgenerates training data for training a plurality of fruit imagescorresponding to big data, stores the generated training data in adatabase, and analyzes the training data to set features for each fruitincluding the form of the size, shape, color, branch, leaf, or the likeof each fruit based on the fruit determination criterion and train theclassifier Therefore, the deep learning-based fruit recognitionalgorithm may output the classification and prediction results includingthe type and quantity of fruit by the trained classifier when the fruittree images are input as input data.

As illustrated in FIG. 7 , the deep learning-based fruit recognitionalgorithm may be implemented as a deep neural network (DNN) having astructure of an input layer, a hidden layer, and an output layer. Thedeep neural network refers to a system or a network that builds one ormore layers and makes a determination based on a plurality of data. Forexample, the deep neural network may be implemented as a set of layersincluding a convolutional pooling layer, a locally-connected layer, anda fully-connected layer. A convolutional pooling layer or a local accesslayer may be configured to extract features within an image. Thefully-connected layer may determine a correlation between features of animage. In some embodiments, the overall structure may be a convolutionalneural network (CNN) structure which is a structure in which the localconnection layer is connected to the convolutional neural network andthe fully-connected layer is connected to the local connection layer. Inaddition, the deep neural network may be formed in a recurrent neuralnetwork (RNN) structure in which nodes of each layer include edgespointing to the nodes and are connected recursively. The deep neuralnetwork may include various determination criteria (i.e., parameters),and further include new determination criteria (i.e., parameters)through analysis of the input image.

In particular, the convolutional neural network may be implemented as astructure in which a feature extraction layer self-training a featurewith greatest discriminative power from the given image data and aprediction layer training a prediction model to produce highestprediction performance based on the extracted features.

The feature extraction layer may be formed in a structure in which aconvolution layer that applies a plurality of filters to each region tocreate a feature map and a pooling layer that pools a feature mapspatially to extract features invariant to a change in position orrotation are alternately repeated several times. The hidden layer of theCNN can be composed of a combination of the pooling layer and thefully-connected layer as well as the convolutional layer.

The deep learning-based fruit recognition algorithm may extract variouslevels of features from low-level features such as points, lines, andsurfaces to complex and meaningful high-level features. The featuresfinally extracted by repeating the convolutional layer and the poolinglayer may be used for the training and prediction of the classifierbecause classification models such as multi-layer perception (MLP) orsupport vector machine (SVM) are coupled in the form of thefully-connected layer.

FIG. 8 is a flowchart explaining a process of matching location data foreach fruit tree in a fruit quantity measurement method according to anembodiment of the present invention. FIG. 8 is a flowchart explaining aprocess of matching location data for each fruit tree in a fruitquantity measurement method according to an embodiment of the presentinvention, FIG. 9 is a diagram explaining a positioning process of anunmanned aerial vehicle according to an embodiment of the presentinvention, and FIG. 10 is a diagram explaining a process of identifyinga fruit tree using RF received signal strength according to distanceaccording to an embodiment of the present invention.

Referring to FIGS. 8 to 11 , the monitoring server 300 may acquire GPSinformation from the GPS unit 120 installed at the boundary point of thefruit tree region, for example, four outermost point (S11), andcalculate the area of the fruit tree region based on the acquired GPSinformation, establish a flight plan for at least one unmanned aerialvehicle 200 for fruit tree recognition, and allow the unmanned aerialvehicle 200 to be controlled automatically or semi-automaticallyaccording to the flight plan (S12).

In this case, the flight plan may include a flight start/end point,measurement time for acquiring RF signals or images for each fruit tree,an expected collision point between unmanned aerial vehicles, avoidancetime or route or flight time schedule information at the expectedcollision point, and the like.

The unmanned aerial vehicle 200 moves in the air above the first fruittree based on the flight plan (S13) and receives the RF signal from theRF transmitter 110 attached to the first fruit tree (S14).

As illustrated in FIG. 9 , the unmanned aerial vehicle 200 uses the RGBcamera 212 to recognize a position that does not overlap with thecorresponding fruit tree T2 and the neighboring fruit trees T1 and T3 inorder to move to an upper portion of a preset fruit tree T2. This mayprevent the unmanned aerial vehicle 200 from obtaining redundantinfrared images of the preset fruit tree T2 and the neighboring fruittrees T1 and T3 in the case of a densely planted fruit tree region.

In addition, as illustrated in FIGS. 10 and 11 , the unmanned aerialvehicle 200 located above the fruit trees may receive uniqueidentification RF signals transmitted from several fruit trees T1, T2,T3, . . . TN, as illustrated in FIG. 11 , the received signal strengthindicator (RSSI) for a plurality of RF signals may be measured and thesignal strength for a plurality of RF signals may be measured to selectthe RF signal with the closest distance, that is, the uniqueidentification RF signal with the highest signal strength, and it may berecognized that the unmanned aerial vehicle 200 is located on the top ofthe fruit tree corresponding to the selected unique identification RFsignal (S15).

The monitoring server 300 acquires matching information between an RFsignal RF #1 and GPS information GPS #1 of a first fruit tree, andstores location data of the first fruit tree (S16).

In this way, the monitoring server 300 matches and stores location datafor each fruit tree from the first fruit tree to an N-th fruit tree,which is the last fruit tree, using the GPS information and RF signalreceived from the unmanned aerial vehicle 200 (S17 and S18).

FIG. 12 is a flowchart explaining a fruit quantity measurement methodaccording to an embodiment of the present invention, FIG. 13 is anexemplary view illustrating an infrared image acquired through anunmanned aerial vehicle according to an embodiment of the presentinvention, FIG. 14 is an exemplary diagram illustrating a process ofdetecting a fruit area in an infrared image of FIG. 13 , and FIG. 15 isan exemplary view illustrating a fruit counting process based on thefruit region detected in FIG. 14 .

As illustrated in FIG. 12 , the monitoring server 300 moves the unmannedaerial vehicle 200 to the fruit tree region and moves the unmannedaerial vehicle 200 to the air above the N-th fruit tree (S21 and S22).In this case, the monitoring server 300 may accurately move the unmannedaerial vehicle 200 to the top of the N-th fruit tree using thepre-stored location data.

The unmanned aerial vehicle 200 may receive the RF signals of the N-thfruit tree (S23), select an RF signal having the largest signalintensity among the received RF signals, and then match the selected RFsignal with the RF signal of the pre-stored location data to determinewhether or not they match, thereby confirming whether the N-th fruittree is correct (S24). In this case, the unmanned aerial vehicle 200 mayrecognize and determine a flight altitude position for acquiring anon-overlapping image between corresponding fruit trees with an RGBcamera in the sky above the fruit trees.

When the N-th fruit tree is identified, the infrared camera 211 of theunmanned aerial vehicle 200 is used to acquire an infrared image (S250),perform fruit recognition processing of the N-th fruit tree (S26), andmeasure and store the number of fruits (S27).

As illustrated in FIGS. 13 to 15 , the monitoring server 300 inputs aninfrared image of a fruit tree (e.g., a citron fruit tree) to the deeplearning-based fruit recognition algorithm, recognizes a fruit throughthe classifier to detect a fruit region, and performs fruit counting inthe detected fruit region.

When the fruit counting from the first fruit tree to the last fruit treeis completed, the monitoring server 300 may store fruit countinginformation for each fruit tree in the fruit tree region and total fruitcounting information, and provide the fruit counting according to arequest of the user terminal 400 (S28 and S29).

The monitoring server 300 may not only provide location data for eachfruit tree and fruit counting information for each fruit tree, but mayalso use the location data for each fruit tree and fruit countinginformation for each fruit tree to provide information on various fruittrees such as a distribution (number of mature trees) of fruit trees inthe fruit tree region and arrangement information, income calculationinformation according to fruit counting information, productioninformation per unit of mature tree area, production forecastinformation based on weather forecast information or accumulatedproduction information, and fruit tree planting plan.

Although each process is described as sequentially executed in FIGS. 8and 12 , this is merely an example of the technical idea of oneembodiment of the present invention. Since those skilled in the art towhich an embodiment of the present invention pertains may change andexecute the sequence illustrated in each drawing within the range notdeparting from the essential characteristics of the embodiment of thepresent invention, and one or more of each process may be applied inparallel with various modifications and variations, FIGS. 8 and 12 arenot limited to a time-series sequence.

Meanwhile, the processes illustrated in FIGS. 8 and 12 may beimplemented as a computer readable code in a computer readable recordingmedium. The computer-readable recording medium may include all kinds ofrecording apparatuses in which data that may be read by a computersystem are stored. That is, the computer-readable recording mediumincludes storage media such as magnetic storage media (e.g., a read-onlymemory (ROM), a floppy disk, a hard disk, etc.) and optically readablemedia (e.g., a compact disc read-only memory (CD-ROM), a digital videodisc (DVD), etc.). In addition, the computer readable recording mediamay be distributed in computer systems connected to each other through anetwork, such that the computer readable codes may be stored andexecuted in the computer readable recording media in a distributedscheme.

The spirit of the present embodiment is illustratively describedhereinabove. It will be appreciated by those skilled in the art to whichthe present embodiment pertains that various modifications andalterations may be made without departing from the essentialcharacteristics of the present embodiment. Accordingly, the presentembodiments are not to limit the spirit of the present embodiment, butare to describe the spirit of the present disclosure. The technical ideaof the present embodiment is not limited to these embodiments. The scopeof the present disclosure should be interpreted by the following claims,and it should be interpreted that all the spirits equivalent to thefollowing claims fall within the scope of the present disclosure.

This patent is the result of a study conducted with the support of theKorea Institute for Advancement of Technology with financial resourcesfrom the Korean government (Ministry of Trade, Industry and Energy) in2021 (Unique Project Number 1415176969, Detailed Project Number:P0014718, Project name: Smart farming demonstration spread project using5G-based drones).

This patent is the result of a study conducted with the support of theKorea Evaluation Institute of Industrial Technology with financialresources from the Korean government (Ministry of Trade, Industry andEnergy) in 2022 (Unique Project Number 1415181149, Detailed ProjectNumber: 20018828, Project name: Development of optical filter forwavelength control and light source module for lighting device)

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. § 119 to Korean PatentApplication No. 10-2021-0033932, filed on Mar. 16, 2021, in the KoreanIntellectual Property Office, the disclosure of which is incorporatedherein by reference in its entirety. In addition, if this patentapplication claims priority for the same reason as above for countriesother than the United States, all the contents are incorporated intothis patent application as references.

What is claimed is:
 1. A fruit quantity measurement system comprising:an unmanned aerial vehicle configured to receive GPS information about afruit tree region of a predetermined area and an RF signal transmittedfrom an RF transmitter installed for each fruit tree and provide fruittree images captured using at least one image sensor while flying basedon a predetermined flight plan over the fruit tree region; and amonitoring server configured to be connected to the unmanned aerialvehicle through a communication network to receive GPS information andan RF signal from the unmanned aerial vehicle, match and store locationdata for each fruit tree, analyze the fruit tree image, measure a numberof fruits for each fruit tree, store fruit counting information, andprovide the stored location data or fruit counting information for eachfruit tree according to a request from a user terminal authorized inadvance.
 2. The fruit quantity measurement system of claim 1, furthercomprising: at least one or more GPS units configured to be installed ata boundary of the fruit tree region to provide GPS information.
 3. Thefruit quantity measurement system of claim 1, wherein the unmannedaerial vehicle includes one or more cameras using an RGB sensor and aninfrared sensor.
 4. The fruit quantity measurement system of claim 1,wherein the monitoring server performs a deep learning-based fruitrecognition algorithm that extracts fruit regions from the fruit treeimage, extracts features from the extracted fruit regions, and thenprovides fruit classification and prediction results based on theextracted features.
 5. The fruit quantity measurement system of claim 4,wherein the deep learning-based fruit recognition algorithm generatesand stores training data for training a plurality of fruit imagescorresponding to big data, trains a classifier by setting the featuresof each fruit, including a size, a shape, and a color of each fruit, asfruit determination criteria through the analysis of the stored trainingdata, and outputs the fruit classification and prediction resultsincluding a type and quantity of fruit for the fruit tree image input bythe trained classifier.
 6. A fruit quantity measurement method performedby a monitoring server that performs a fruit tree monitoring functionfor a fruit tree region, the fruit quantity measurement methodcomprising: a) acquiring GPS information about the fruit tree region,and generating a flight plan for at least one unmanned aerial vehiclefor fruit tree recognition based on the acquired GPS information; b)transmitting a flight control signal to the unmanned aerial vehicle formoving the unmanned aerial vehicle to air above a predetermined fruittree based on the flight plan of the unmanned aerial vehicle; c)identifying a predetermined fruit tree by receiving an RF signaltransmitted from an RF transmitter installed in the fruit tree throughthe unmanned aerial vehicle, and matching and storing uniqueidentification information of the predetermined fruit tree and locationdata; and d) receiving fruit tree images through the unmanned aerialvehicle, analyzing the fruit tree images, measuring the fruit quantityfor each fruit tree, and storing fruit counting information.
 7. Thefruit quantity measurement method of claim 6, wherein step b) includesrecognizing and determining, by the unmanned aerial vehicle, a flightaltitude position for acquiring non-overlapping images betweencorresponding fruit trees with an RGB camera in a sky above the fruittrees.
 8. The fruit quantity measurement method of claim 6, furthercomprising: e) providing stored location data or fruit countinginformation for each fruit tree according to a request of a userterminal authorized in advance.
 9. The fruit quantity measurement methodof claim 6, wherein the unmanned aerial vehicle includes one or morecameras using an RGB sensor and an infrared sensor, and provides a fruittree image including an RGB image and an infrared image through thecamera.
 10. The fruit quantity measurement method of claim 6, wherein,in step d), a deep learning-based fruit recognition algorithm extractsfruit regions from the fruit tree image, extracts features from theextracted fruit regions, and then provides fruit classification andprediction results based on the extracted features.
 11. The fruitquantity measurement method of claim 10, wherein the deep learning-basedfruit recognition algorithm generates and stores training data fortraining a plurality of fruit images corresponding to big data, trains aclassifier by setting the features of each fruit, including a size, ashape, and a color of each fruit, as fruit determination criteriathrough the analysis of the stored training data, and outputs the fruitclassification and prediction results including a type and quantity offruit for the fruit tree image input by the trained classifier.
 12. Amonitoring server, comprising: a communication unit configured toperform a communication function with an unmanned aerial vehicle flyingover a fruit tree region or a user terminal authorized in advance; and amemory in which a program for performing a fruit quantity measurementmethod for the fruit tree region is recorded; and a processor configuredto execute the program, wherein the processor is connected to theunmanned aerial vehicle through a communication network by execution ofthe program to receive GPS information and an RF signal from theunmanned aerial vehicle, match and store location data for each fruittree provided from the unmanned aerial vehicle, analyze the fruit treeimage, measure a number of fruits for each fruit tree, store fruitcounting information, and provide the stored location data or fruitcounting information for each fruit tree according to a request from theuser terminal.